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Abstract. The successful implementation of the next generation infrastructure systems requires solid understanding of their 
technical, social, political and economic aspects along with their interactions. The lack of historical data that relate to the long-term 
planning of complex systems introduces unique challenges for decision makers and involved stakeholders which in turn result in 
unsustainable systems. Also, the need to understand the infrastructure at the societal level and capture the interaction between 
multiple stakeholders becomes important. This paper proposes a methodology in order to develop a holistic approach aiming to 
provide an alternative subject-matter expert (SME) elicitation and data collection method for future sociotechnical systems. The 
methodology is adapted to Next Generation Air Transportation System (NextGen) decision making environment in order to 
demonstrate the benefits of this holistic approach. 


1.0 INTRODUCTION 

The world is heavily dependent on various 
critical infrastructures in areas like 
transportation, power, communication, 
water, energy, etc. Today’s critical 
infrastructures are large-scale socio- 
technical systems, comprised of multiple 
components, involving various stakeholders, 
technologies, policies and social factors [1], 
In the recent years, numerous 
sociotechnical systems started to undergo a 
series of transitions. The definition of 
system transition is given as “a long-term 
fundamental change (irreversible, high- 
impact and of high-magnitude) in the 
cultures (mental maps, perceptions), 
structures (institutions, infrastructures and 
markets), and practices (use of resources) 
of a societal system” [1], In other words, the 
transition includes “a structural change in 
both technical and social subsystems” [2], 

The planning and implementation phases of 
such large-scale infrastructure transitions 
require close monitoring of performance 
parameters like safety, efficiency, and 
sustainability. Ensuring that infrastructure 
transition reveals a safer and more 
sustainable system became a major 
challenge for the society [3], In order to do 
so, decision makers often need to test 
various strategies and perform analyses to 


characterize risk and other parameters. 
However past strategies and historical data 
regarding previous infrastructure systems 
are no longer adequate for next generation 
infrastructure system design due to (1) 
previous systems evolved via incremental 
changes and system improvements which 
lead them to be unsustainable (i.e. 
congestion, energy shortage, air 
transportation delays, etc.) and (2) previous 
infrastructures were made to last, robust but 
resistant to change [1, 4]. The lack of 
empirical data causes decision makers to 
heavily rely on expert opinions for next 
generation infrastructure planning. 

1.1.1 Lack of Data and Expert 
Elicitation 

The future status of man-made systems like 
energy, transportation, warfare, agriculture, 
and other infrastructure cannot be predicted 
over a prolonged time frame. Large-scale 
sociotechnical systems are made up of 
multiple components that involve numerous 
stakeholders, technologies, policies, and 
social factors [1]. Decision makers and 
policy makers often require expert opinion 
to comprehend the complexity and 
uncertainties within such systems. Expert 
elicitation methods typically have been used 
to obtain the necessary data for reliability 
and risk studies for these types of 


technological, environmental, and 
socioeconomic issues [5, 6], Furthermore, 
NextGen will inherently be different from the 
current system. However, the ability to 
predict the future remains limited owing to 
the long-term implementation phase and the 
large number of uncertainties [5], “There 
are no data about the future on which to 
rely. We are challenged to imagine many 
different and possible ‘futures’ as 
humankind seeks to exert its mastery and 
control” [7]. The crucial task is to think 
innovatively and recognize the creative and 
imaginative capacities of each stakeholder. 
The overall goal of the methodology is to 
reduce complexity and uncertainty while 
inventing the future and analyzing the 
respective risk for each alternative scenario 
[7]- 

1.1.2 Sociotechnical Complexity 

The system-wide upgrade of complex 
systems is a challenging undertaking [8], 
The increased complexity adds to the 
diversity of decision maker’s system 
interpretations that can directly alter the 
overall system operation and the decision- 
making processes. Brewer [7] states that 
real-world problems do not exist 
independently of their socio-cultural, 
political, economic, and physiological 
content, and for that reason, an approach 
with multiple perspectives and multiple 
disciplines is necessary to efficiently clarify 
the matter at hand (which is quite 
challenging in practice). The presence of 
multiple actors with frequently divergent and 
conflicting interests can turn large-scale 
infrastructure transitions into wicked 
problems [9], Traditional policy and market 
practices have proven ineffective in dealing 
with problems with a high degree of 
uncertainty regarding future scenarios and 
actor interactions [1]. For this reason, 
creating a methodology that attempts to 
predict decision pathways for future 
systems while accounting for the technical, 
organizational, and contextual complexity of 
the system is necessary [4, 7], 


2.0 TEST CASE 

The goal of this research is to develop a 
methodology that will serve as an aid for 
decision makers who are responsible for 
designing and evaluating scenarios for 
future technological implementations within 
next generation infrastructure systems. As 
previously mentioned, the implementation of 
large system transitions require 
understanding of the multi-layer complexity 
and overcoming the lack of experimental 
data for designing the future phases of the 
system. In order to demonstrate the 
proposed methodology, the planning, 
development, and implementation of the 
Next Generation Air Transportation System 
(NextGen) is used as a test case. The 
following sections will provide insights on 
NextGen. 

2.1 NextGen Overview 

The U.S. National Airspace System (NAS) 
is made up of a number of multifaceted 
elements, including over 800 billion 
passengers and input from more than 

15.000 air traffic controllers to assist 

590.000 pilots onboard 239,000 aircraft that 
take off and land at 20,000 U.S. airports. 
This extremely complex system is closely 
tied to the national economy, contributing 
$1 .2 trillion annually and over 5 percent of 
the gross domestic product while generating 
1 1 million jobs and $369 billion in earnings 
[ 10 ]. 

The delays that currently impact passenger 
travel are forecasted to be even higher in 
the future as the demand for air 
transportation is expected to increase. In 
addition, future airspace is expected to 
accommodate unmanned aircraft systems 
and commercial space vehicles as well. 
Furthermore, the entire system is expected 
to operate within acceptable safety levels 
and environmental impact guidelines [10]. 

To respond to this forecasted increase in 
demand, the Joint Planning and 
Development Office (JPDO) was formed 
during the Bush Administration in 2003. This 
organization is a partnership between public 



and private stakeholders, including the 
Federal Aviation Administration (FAA), the 
Department of Defense (DoD), the 
Department of Homeland Security (DHS), 
NASA, and others [1 1], The JPDO is 
charged with developing concepts, 
architectures, roadmaps, and 
implementation plans for transforming the 
current national Air Transportation System 
(ATS) into NextGen. 

“During the next two decades, demand will 
increase, creating a need for a system that 
(1) can provide two to three times the 
current air vehicle operations; (2) is agile 
enough to accommodate a changing fleet 
that includes very light jets (VLJs), 
unmanned aircraft systems (UASs), and 
space vehicles; (3) addresses security and 
national defense requirements; and (4) can 
ensure that aviation remains an 
economically viable industry” [8], 

2.2 NextGen Challenges 

The complex nature of the NAS, combined 
with numerous operational and 
management challenges, threatens the 
NextGen efforts. Reports from the Office of 
the Inspector General (OIG) reveal that the 
Federal Aviation Administration (FAA) is 
facing difficulties in developing a strategy to 
engage stakeholders, not to mention 
managing and integrating multiple NextGen 
efforts [1 2], Uncertainties and a lack of data 
related to shaping a future aviation system 
also inhibit the ability to employ formal risk 
analysis methods. As a result, SME opinion 
has become the primary source of input for 
the NextGen scenarios, technologies, and 
safety. 

2.3 Need for a Methodology 

In the past, traditional engineering design 
approaches focused primarily on the 
technical requirements. Similarly, traditional 
infrastructure designs were treated like 
traditional engineering problems, causing 
them to be brittle and resistant to 
modernization. However, the next 
generation infrastructures must be treated 
differently because of their complex nature 


[13], The need to understand the 
infrastructure at the societal level and 
capture the interaction between the 
technical, political, and economic factors 
becomes more important [4], Traditional 
engineering design methods are used in 
concert with serious gaming approaches to 
create a holistic decision-making 
methodology. The goal of the proposed 
methodology is to enable decision makers 
and researchers to gather information in 
regard to NextGen safety values. Toward 
that purpose, various tools and techniques 
are employed collectively here to create a 
methodology that can be used as an 
alternative to conventional expert elicitation 
techniques (i.e. Delphi Method, Nominal 
Group Technique, brainstorming, etc.) for 
complex systems with multiple 
stakeholders. 

3.0 PROPOSED METHODOLOGY 

As discussed above, the methodology for 
estimating risks within a future system will 
combine various approaches. Because the 
air transportation system includes extensive 
interactions between multiple stakeholders, 
which can be difficult to track, and because 
of the lack of historical data, SMEs from 
diverse backgrounds are the main source of 
data for this study [6], Aviation safety within 
NextGen is measured by using the 
probability number method. Conventional 
numerical and qualitative expert elicitation 
techniques provide the gaming data that are 
necessary to construct the scenarios, 
alternatives, attributes, and so on. 
Commercial-off-the-shelf software packages 
(i.e., Logical Decisions for Windows® and 
Precision Tree® by Palisade Corp.) are also 
used to rank future technologies in order to 
support SME opinions before and during the 
gaming cycle. Fig. 1 provides an overview 
of the methodology. The various tools and 
techniques are described in more detail in 
subsequent sections. 
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Figure 1. Serious gaming methodology: high-level 
architecture 


3.1 Serious Gaming and 
Infrastructure Design 

The use of gaming and simulation 
techniques as a formal approach to strategy 
making has gained wide acceptance, as 
evidenced by the frequency of occurrence 
within mainstream strategy literature [14], 
Gaming methods (or soft systems thinking) 
have become an alternative to formal 
complexity modeling techniques like 
systems dynamics and operations research. 
These techniques have been successfully 
applied to well-structured problems; 
however, when employed on ambiguous 
and often ill-structured and complex 
systems, their contribution has been limited 
because adequate theory and empirical 
data were absent [14]. 

Serious gaming methods are able to provide 
decision makers with an environment in 
which the totality of the system and its 
dynamics are present. With a holistic 
approach that includes the wide-range of 
perspectives, skills, information, and mental 
models of the involved parties, the quality of 
the decision-making environment increases 
dramatically [13, 14]. Unlike hard-system 
methods, the gaming and simulation 
approach is quite flexible and easily 
adaptable to other quantitative methods, 
scenarios, and computer models [16]. 

Policy gaming methods can help both 
participants and modelers understand the 
big picture and identify critical elements of 


the complex problem at hand. Because of 
the iterative and experimental nature of 
these gaming and simulation environments, 
participants are able to test different 
approaches within both a safe environment 
and a condensed timeframe [15]. 

According to Duke [17], a typical complex 
real-world situation has the following 
characteristics: it contains numerous 
variables in interaction; no realistic basis 
exists for quantification of these variables or 
their interactions; and no proven conceptual 
model or precedent exists on which action 
decisions can be based. Complex systems 
are also typified by a sociopolitical context 
of decision-making, where the actions of the 
various “players” may be idiosyncratic or 
irrational; furthermore, the decisions are 
irreversible and the results are not generally 
fully understood until well into the future 
[17]. NextGen fits this model, as a complex 
real-world air-transportation system that will 
undergo a full-scale transformation and that 
contains numerous stakeholders with often 
conflicting agendas, including those of the 
general public [18], The gaming context 
may help capture the organizational and 
behavioral dynamics of the decision-making 
process and ultimately yield a more realistic 
problem solution. 

The following section provides insight on the 
probability number method (PNM), which is 
used as the backbone for the NextGen risk 
calculation method. 

3.2 Probability Number Method 

The PNM was created through a joint effort 
between the International Atomic Energy 
Agency (IAEA) and several United Nations 
organizations. The method was developed 
as an affordable solution to quickly 
determine the risks that are associated with 
handling, storing, processing, and 
transporting hazardous materials. The 
methodology is supported by an extensive 
database that includes the various factors 
that impact the risks, including types of 
substances (i.e. , flammable, toxic, or 
explosive gases or liquids), safety 








precautions, population densities, 
environmental factors, and so on [19], The 
average accident scenario contains only 
rough estimates because the purpose of the 
methodology is to serve as a decision- 
making aid that enables risk ranking and 
prioritization for further analysis. Dr. Adrian 
Gheorghe was a part of the development 
team for the PNM and brings his expertise 
to decoding, modifying, and adapting the 
probability number method to the NextGen 
system. 

3.2.1 Probability Number 

Within the PNM, the probability of the 
occurrence of a certain accident is 
calculated via a dimensionless probability 
number A/, which is then transformed into an 
actual probability. The probability number 
can be adjusted or updated based on 
various correcting factors. The risk is 
defined as the product of the consequences 
and the probabilities of unwanted outcomes 
(i.e. , hazardous events). 

3.2.2 Adapted Consequences and 
Probabilities 

The PNM defines risk as the product of the 
probability of an accident and its respective 
consequences, calculated separately. 

Within the NextGen framework, the 
consequences are defined as fatal aviation 
accidents (i.e., accidents per 100,000 flight 
hours). The probability of an accident that 
involves a passenger fatality is calculated in 
the following manner. An average 
probability number that represents the base 
assumption is determined; then, this 
number is adjusted by using correcting 
factors. These factors represent 
technological improvements and other 
enablers that are planned within the 
NextGen framework, namely, runway 
safety, aircraft reliability, icing, turbulence 
mitigation, weather, and airborne collision 
avoidance. The adjusted accident 
probability N is then converted into a 
frequency of occurrence. 


The probability of occurrence and the 
consequence factors are inputted into the 
FAA’s Risk Matrix (Fig. 2). The initial 
conditions for the risk are determined by the 
averaged accident data (2000-2009) which 
are obtained from NTSB website 1 . The 
average severity of aviation accidents is 
0.291 fatalities/100,000 flight-hours or 
severity classification “Minor, 4”; meanwhile 
the probability of such an accident is 
0.208/100,000 flight-hours, indicating a 
“Remote, C” likelihood category. Departing 
from the values above, the current aviation 
risk is determined as “Low Risk, green”. 
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Figure 2. FAA risk matrix. 

The PNM was chosen to be the backbone 
for the risk estimation engine in this 
application as a result of its intuitive 
structure and ease of expandability. The 
main components of the PNM, namely, the 
consequences, probabilities, and risk 
outcomes, are incorporated into the 
NextGen Safety Assessment Methodology 
and fused with the policy gaming effort. 

3.3 Software Add-Ons 

The selected gaming platform enables the 
integration of additional methods and 
techniques, which allows the methodology 
to remain flexible and expandable. This in 
turn ensures a more thorough 


1 http://www.ntsb.gov/aviation/Stats.htm 




representation of the air transportation 
infrastructure. Two commercial-off-the-shelf 
software solutions have been embedded 
into the methodology to enhance the 
ranking and prioritization of enablers and 
other damage indicators. The Logical 
Decisions for Windows® software helps the 
prioritization and ranking of NextGen safety- 
related technological enablers with respect 
to their benefits, costs, implementation 
timelines, and other parameters. The 
Precision Tree® software is used to collect 
and further analyze the data that are 
obtained from the gaming exercise. 

4.0 GAMING SEQUENCE 

The NextGen safety assessment 
methodology was developed on a serious 
gaming platform that was adopted from the 
play sequence of policy gaming developed 
by Geurts, Duke, and Vermeulen [14]. An 
adapted version of the play sequence, 
which accommodates the NextGen safety 
framework, is given in Fig. 3. The sequence 
is initiated by the presentation of the game 
to the stakeholders; this includes providing 
the game rules, a general overview of 
NextGen goals, and the available 
resources. Stakeholder groups that contain 
participants from various backgrounds are 
formed, and their respective goals are 
established (e.g., the FAA is concerned with 
safety goals, commercial airlines with 
economic goals, and so on). The groups 
are asked to evaluate and select from the 
list of technological advancements that are 
related to the improvement of safety. 
However, the implementation of each of 
these advancements consumes some of the 
predetermined limited resources. 
Stakeholders with conflicting agendas must 
come to a consensus on certain decisions. 
Following these discussions, the decisions 
for each time step are entered into the risk 
simulation mechanism (based on the PNM) 
and the updated NAS risk values are 
calculated iteratively for the next three time 
steps, until year 2025. The gaming exercise 
is concluded with the debriefing and 
discussions in order to create the foundation 
for the data gathering and analysis. 


Scenario Presentation: National Airspace System in 
2025 
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Figu re 3. NextGen safety assessment gaming 
sequence. 

4.1 Stakeholders and Game Rules 

One of the most productive outcomes of the 
policy gaming exercise is that the 
participants are able to interact based on 
the problem at hand. The “safe” 
environment allows participants to create 
and analyze the system complexity while 
communicating various aspects of the issue 
among the stakeholders [7]. In order to 
model such a dynamic environment, a 
simplified list of involved stakeholders and 
engagement rules was developed. 

Stakeholder interactions can be based on 
rigid rules, free-form rules or combination of 
the two. Rigid, rule-based gaming is well 
suited for structured environments, such as 
military gaming where specific rule sets that 
can be formalized by mathematical or 
computational methods are used. However, 
for social arenas that include both public 
and intense stakeholder interactions without 
firm rules, free-form gaming, which relies on 
game rules, is more suitable. Free-form 
games enable the participants to challenge, 
modify, and improve the positions, objects, 
and rules during the game play. However, 
the process must be carefully monitored by 
a control team of experts who act as 
referees or game directors [16]. 

Within the scope of this project, the primary 
goal was to provide insight into the future of 
NAS safety and data gathering in regard to 
future systems. Thus, a combination of rigid 
and free-form gaming rules was employed. 

The goal of the game was to simulate the 





aviation safety values within a given time 
frame, while taking into consideration 
technological constraints (i.e., cost versus 
benefit, feasibility, mixed equipage, and so 
on) and behavioral concerns (i.e., 
information overload to pilots, controllers, 
early technology adopters, and so on). 

4.2 Outcomes 

Throughout the gaming effort, the 
discussions and negotiations that occurred 
between participants with opposing 
agendas were important and can be used to 
develop different problem-solving 
approaches. The gaming exercise serves 
both as an individual and a collective 
learning platform for the stakeholders, 
leading to an overall elevated level of 
knowledge across the system. The 
individual learning took place during the 
decision-making process during which each 
stakeholder group represents their 
respective point of view. The acquired 
awareness that was gained in regard to the 
overall system complexity will ultimately 
improve the value of the expert elicitation. 
Furthermore, the presence of realistic 
interactions among the players yielded data 
that can be used in the testing and 
evaluation of NextGen-related technologies 
in the future [20], In addition to the 
individual learning, the collective learning 
(or the organizational learning) provided 
valuable insight that relates to the problems 
that were discussed (i.e., NextGen aviation 
safety values). 

One of the most tangible outcomes of the 
gaming exercise was the 2025 NAS safety 
values with respect to the FAA’s Risk Matrix 
(Fig. 2) acceptability measures. The 
intermediate risk values that were obtained 
during the technology implementation phase 
(i.e., the next 15 years) were elicited under 
the same assumptions. The cumulative 
effect of various safety-related technological 
implementations can be examined, which 
will enable decision makers to identify 
technologies or areas that require further 
analysis and understanding. 


5.0 CONCLUSIONS 

The planning and implementation of next 
generation infrastructure transitions are 
challenging due to their complex nature and 
the lack of historical data. This paper 
proposes the use of simulation and gaming 
methods as a platform for evaluating and 
generating necessary data for designing 
future infrastructure systems. The Next 
Generation Air Transportation System 
(NextGen) decision making environment is 
used as a test-bed to demonstrate the 
developed methodology. 

Subject-matter-expert opinions were heavily 
relied upon to develop the gaming 
components, to decide on the participants, 
and finally, to evaluate the validity of the 
framework. Conventional risk calculation 
methods and commercial-off-the-shelf 
software capabilities were integrated to 
provide system-level overview and risk 
analysis as a decision-making tool. One of 
the most prominent contributions of the 
gaming exercise was its ability to aggregate 
the perspectives of multiple stakeholders 
with varying agendas, while calculating the 
effectiveness of future NextGen safety 
enablers. The gaming environment 
promotes individual and collective learning 
across the system, allowing subject-matter 
experts to express their opinions for a more 
thorough and accurate modeling of the 
future infrastructure. 
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